Methods and Systems for Educational On-Line Methods

ABSTRACT

A Double-Loop Mutual Assessment (DLMA) method may assess complex, non-objective, content. For example, a DLMA method can use formative and summative peer assessment to generate textual feedback and/or numeric success metrics. One or more DLMA methods can be used in any number of situations. For example, one or more DLMA methods may be used in online courses, in-person courses, blended courses, written submissions, consumer assessment of products and/or services, performance evaluation, assessing individual contributions to group projects, and/or other tasks.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. §119(e) to U.S.Provisional Application No. 61/646,640, filed May 14, 2012, entitled“Methods and Systems for Educational On-Line Methods,” the entirety ofwhich is hereby incorporated reference.

FIELD OF THE INVENTION

The present disclosure relates generally to methods and systems foreducational on-line methods and more particularly relates to assessingoutcomes of complex task competencies among participants.

BACKGROUND

Historically, the ability to assess and empirically demonstratecompetencies, attainment, and/or improvement of an individual within agiven population has been difficult. Similarly, the ability to assessand empirically demonstrate competencies, attainment, and/or improvementof groups within a population has also been difficult. Systems andmethods that enable competencies, attainment, and/or improvement of anindividual within a given population and/or a group within a givenpopulation to be assessed and/or empirically demonstrated would beadvantageous. In addition, systems and methods that improve complex taskperformance, mitigate deficiencies in existing peer assessment systems,and/or enable large-scale evolutions involving one or more participantswould be advantageous.

SUMMARY

Embodiments of the present invention provide systems and methods forassessing outcomes of complex task competencies. For example, in oneembodiment of the present invention, a Double-Loop Mutual Assessment(DLMA) method is usable as a peer assessment tool. In an embodiment, oneor more DLMA methods can help to assess outcomes of complex taskcompetencies, such as expertise, among participants. In one embodiment,a DLMA method uses both formative and summative peer assessments togenerate feedback and success metrics. For example, a DLMA may providetextual feedback and numerical scores for one or more participants. DLMAmethods can be designed to be and may be applicable to any number ofsettings. For example, in various embodiments, one or more DLMA methodsmay be used to qualitatively grade courses. A course may be an onlinecourse or an in-class course, or a combination thereof. In otherembodiments, one or more DLMA methods can be used to select academicjournal articles and/or conference submissions. As another example, oneor more DLMA methods may be used to assess individual performance on aseries of complex tasks in social settings, assess individualcontributions to group projects, evaluate an individual or group'sperformance, assess products and/or services for one or more consumers,assess collaborative environments such as a collaborative onlineencyclopedia, build competency-based social systems of learning such ascreative writing or photography or art courses, and/or numerous othercomplex tasks.

These illustrative embodiments are mentioned not to limit or define theinvention, but rather to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description,which provides further description of the invention. Advantages offeredby various embodiments of this invention may be further understood byexamining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more examples ofembodiments and, together with the description of example embodiments,serve to explain the principles and implementations of the embodiments.

FIG. 1 is a DLMA workflow according to an embodiment of the presentinvention;

FIG. 2 is a block diagram depicting an exemplary requesting or receivingdevice according to an embodiment;

FIG. 3 is a system diagram depicting exemplary computing devices in anexemplary computing environment according to an embodiment;

FIG. 4 illustrates a method of implementing a DLMA workflow according toan embodiment of the present invention;

FIG. 5 illustrates a workflow schema of a Double-Loop Mutual Assessment(DLMA) Peer Assessment Information System (PAIS) according to anembodiment of the present invention

FIG. 6 illustrates a logical relationship of algebraic models of a DLMAscore generation process according to an embodiment of the presentinvention; and

FIG. 7 illustrates an exemplary dyad formation in closed groups and onnetworks according to an embodiment of the present invention.

DETAILED DESCRIPTION

Example embodiments are described herein in the context of assessingoutcomes of complex task competencies among participants. Those ofordinary skill in the art will realize that the following description isillustrative only and is not intended to be in any way limiting. Otherembodiments will readily suggest themselves to such skilled personshaving the benefit of this disclosure. Reference will now be made indetail to implementations of example embodiments as illustrated in theaccompanying drawings. The same reference indicators will be usedthroughout the drawings and the following description to refer to thesame or like items.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another.

Overview

One or more DLMA methods may help to assess outcomes of one or morecomplex tasks. In one embodiment, a complex task is characterized byvarious combinations of complexity attributes. For example, complexityattributes may include, but are not limited to, such attributes asoutcome multiplicity, solution scheme multiplicity, conflictinginterdependence, and solution scheme and/or outcome uncertainty. Invarious embodiments, complex tasks can include, but are not limited to,writing essays, creating compositions, and/or producing academicarticles.

In an embodiment, a DLMA method is based on a workflow that facilitiesformative assessment and/or summative assessment. In one embodiment,formative assessment provides a set of formal and/or informal evaluationprocedures with the intent of improving a subject's competencies throughbehavior modification. For example, formative assessment may provideresults using qualitative feedback. In an embodiment, summativeassessment is intended to measure a subject's attainment at a particulartime. For example, summative assessment may provide externalaccountability in the form of a score and/or a grade. In variousembodiments, one or more modes of DMLA may be used. In one embodiment, amode of DLMA is a type of scale that is used for summative assessment.For example, ranking and/or rating are examples of modes of DMLAaccording to an embodiment. In one embodiment, ranking provides asummative assessment mode based on a relative scale, forceddistribution, and/or another suitable scale and/or distribution. In anembodiment, rating provides a summative assessment mode based on anabsolute-scale, Likert-scale, or another suitable scale and/ordistribution.

In assessing outcomes of one or more complex tasks, peer assessments maybe involved. In one embodiment, a peer assessment is an arrangement ofassessment in which subjects consider the products and/or outcomes ofpeer subjects of similar status. For example, subjects may consider theamount, level, value, worth, quality, success, other factors, or acombination thereof of the products and/or outcomes of peer subjects. Inembodiments, feedback is provided as part of peer assessment. In oneembodiment, the feedback provides an instance of formative assessmentwhich is given by one peer to another. For example, a subject mayprovide a written statement regarding the quality of another subject'sessay. In another embodiment, feedback, such as gauging and/or feedbackevaluation, provides an instance of summative assessment given by onepeer to another peer.

Illustrative DLMA Workflow

FIG. 1 is a DLMA workflow according to an embodiment of the presentinvention. In the embodiment shown in FIG. 1, a classroom of studentsare divided into groups of six students. Each group is given anassignment to complete. For example, a group may be assigned an articleto read, perform a case analysis on, and draft an essay having 750 wordsor less regarding the article and case analysis. The assignment can bethe same for each group or one or more of the groups can have differentassignments. Each student in each group completes the assignment. Forexample, each student in the group may write an essay 100. The essaysthat the students write can be submitted through an online website toone or more databases. The students then evaluate the essays of otherstudents in their group 110. For example, if a group contains sixstudents, then one student in the group may evaluate the essays of theother five students in the group. The student may rank the otherstudents' essays from best to worst and may provide written feedbackregarding the strengths and weaknesses of the other students' essays120. A student's evaluations of the other students' essays in the groupcan be submitted through the online website and stored in one or moredatabases. The students in the group then receive feedback regardingtheir essay and scores for the essays are generated 130. The students inthe group then evaluate the evaluations that they received from theother students in the group and score the evaluations 140. The rankingsof the evaluations can be submitted through the online website and maybe stored in one or more databases. This process may be repeatedmultiple times. For example, the same groups may be given a secondassignment. As another example, the students in the classroom may bedivided into new groups and given a second assignment. The results of asingle assignment and/or multiple assignments can be evaluated todetermine a ranking for the students. In addition, overall feedback canbe provided to the students. For example, a particular student may beprovided feedback indicating that he or she is writing essays very wellbut is ranking poorly in providing feedback for other students' essays.

This illustrative example is given to introduce the reader to thegeneral subject matter discussed herein. The invention is not limited tothis example. The following sections describe various additionalnon-limiting embodiments and examples of devices, systems, and methodsfor content- and/or context-specific haptic effects.

Illustrative Device

FIG. 2 is a block diagram depicting an exemplary requesting or receivingdevice according to an embodiment. For example, in one embodiment, thedevice 200 may be a web server, such as the web server 350 shown in FIG.3. In other embodiments, device 200 may be a client device, such as theclient devices 320-340 shown in FIG. 3. In various embodiments, device200 may be a tablet computer, desktop computer, mobile phone, personaldigital assistant (PDA), or a sever such as a web server, media server,or both.

As shown in FIG. 2, the device 200 comprises a computer-readable mediumsuch as a random access memory (RAM) 210 coupled to a processor 220 thatexecutes computer-executable program instructions and/or accessesinformation stored in memory 210. A computer-readable medium maycomprise, but is not limited to, an electronic, optical, magnetic, orother storage device capable of providing a processor withcomputer-readable instructions. Other examples comprise, but are notlimited to, a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM,RAM, SRAM, DRAM, CAM, DDR, flash memory such as NAND flash or NOR flash,an ASIC, a configured processor, optical storage, magnetic tape or othermagnetic storage, or any other medium from which a computer processorcan read instructions. In one embodiment, the device 200 may comprise asingle type of computer-readable medium such as random access memory(RAM). In other embodiments, the device 200 may comprise two or moretypes of computer-readable medium such as random access memory (RAM), adisk drive, and cache. The device 200 may be in communication with oneor more external computer-readable mediums such as an external hard diskdrive or an external DVD drive.

The embodiment shown in FIG. 2, comprises a processor 220 which executescomputer-executable program instructions and/or accesses informationstored in memory 210. The instructions may comprise processor-specificinstructions generated by a compiler and/or an interpreter from codewritten in any suitable computer-programming language including, forexample, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, andActionScript®. In an embodiment, the device 300 comprises a singleprocessor 220. In other embodiments, the device 200 comprises two ormore processors.

The device 200 as shown in FIG. 2 comprises a network interface 230 forcommunicating via wired or wireless communication. For example, thenetwork interface 230 may allow for communication over networks viaEthernet, IEEE 802.11 (Wi-Fi), 802.16 (Wi-Max), Bluetooth, infrared,etc. As another example, network interface 230 may allow forcommunication over networks such as CDMA, GSM, UMTS, or other cellularcommunication networks. The device 200 may comprise two or more networkinterfaces 230 for communication over one or more networks.

The device 200 may comprise or be in communication with a number ofexternal or internal devices such as a mouse, a CD-ROM, DVD, a keyboard,a display, audio speakers, one or more microphones, or any other inputor output devices. For example, the device 200 shown in FIG. 2 is incommunication with various user interface devices 240 and a display 250.Display 250 may use any suitable technology including, but not limitedto, LCD, LED, CRT, and the like.

Device 200 may be a server, a desktop, a personal computing device, amobile device, or any other type of electronic devices appropriate forproviding one or more of the features described herein.

Illustrative System

FIG. 3 illustrates a system diagram depicting exemplary computingdevices in an exemplary computing environment according to anembodiment. The system 300 shown in FIG. 3 includes three clientdevices, 320-340, and a web server 350. Each of the client devices,320-340, and the web server 350 are connected to a network 310. In thisembodiment, each of the client devices, 320-340, is in communicationwith the web server 350 through the network 310. Thus, each of theclient devices, 320-340, can send requests to the web server 350 andreceive responses from the web server 350 through the network 310.

In an embodiment, the network 310 shown in FIG. 3 facilitatescommunications between the client devices, 320-340, and the web server350. The network 310 may be any suitable number or type of networks orlinks, including, but not limited to, a dial-in network, a local areanetwork (LAN), wide area network (WAN), public switched telephonenetwork (PSTN), the Internet, an intranet or any combination ofhard-wired and/or wireless communication links In one embodiment, thenetwork 310 may be a single network. In other embodiments, the network310 may comprise two or more networks. For example, the client devices320-340 may be connected to a first network and the web server 350 maybe connected to a second network and the first and the second networkmay be connected. Numerous other network configurations would be obviousto a person of ordinary skill in the art.

A client device may be any device capable of communicating with anetwork, such as network 310, and capable of sending and receivinginformation to and from another device, such as web server 350. Forexample, in FIG. 3, one client device may be a tablet computer 320. Thetablet computer 320 may include a touch-sensitive display and be able tocommunicate with the network 310 by using a wireless network interfacecard. Another device that may be a client device shown in FIG. 3 is adesktop computer 330. The desktop computer 330 may be in communicationwith a display and be able to connect to the network 310 through a wirednetwork connection. The desktop computer 330 may be in communicationwith any number of input devices such as a keyboard of a mouse. In FIG.3, a mobile phone 340 may be a client device. The mobile phone 340 maybe able to communicate with the network 310 over a wirelesscommunications means such as TDMA, CDMA, GSM, or WiFi.

A device receiving a request from another device may be any devicecapable of communicating with a network, such as network 310, andcapable of sending and receiving information to and from another device.For example, in the embodiment shown in FIG. 3, the web server 350 maybe a device receiving a request from another device (i.e. client devices320-340) and may be in communication with network 310. A receivingdevice may be in communication with one or more additional devices, suchas additional servers. For example, web server 350 in FIG. 3 may be incommunication with another server that encodes or segments, or both,media content from one or more audio or video inputs, or both. In thisembodiment, the web server 350 may store the segmented media files on adisk drive or in cache, or both. In an embodiment, web server 350 may bein communication with one or more audio or video, or both, inputs. Inone embodiment, a web server may communicate with one or more additionaldevices to process a request received from a client device. For example,web server 350 in FIG. 3 may be in communication with a plurality ofadditional servers, at least one of which may be used to process atleast a portion of a request from any of the client devices 320-340. Inone embodiment, web server 350 may be part of or in communication with acontent distribution network (CDN) that stores data related to one ormore media assets.

Illustrative DLMA According to an Embodiment

In one embodiment, a DLMA system is based on the workflow thatfacilitates two interdependent processes: (1) the exchange of essays andfeedback among several subject in a small group or a network thataccommodates a learning dialogue (e.g., formative assessment), and (2)score generating process that ultimately forms a distribution of aperformance metric (e.g., summative assessment).

A DLMA workflow can function as a virtual social system with a certainstructure and relationships. For example, a basic unit of interactionwithin DLMA is a dyad of subjects (i.e., subject i to subject j). Insuch an embodiment, the interaction within the dyad of subjects caninvolve a sequence of reciprocal exchanges for one or more assessedtasks. All or a portion of the sequence of reciprocal exchanges may beanonymous, non-anonymous, or a combination thereof. In one embodiment,the sequence of reciprocal exchanges involves representations of complextask solutions. The representation of a complex task may be referred toas an essay. In one embodiment, an essay comprises an instance of acomplex task outcome being assessed. In another embodiment, a sequenceof reciprocal exchanges includes formative assessment of and/or feedbackto essays. In some embodiments, a sequence of reciprocal exchanges forone or more assessed tasks can include both essays and formativeassessment of and/or feedback to essays.

According to one embodiment, each subject provides a summativeassessment of other peers' essays according to various criteria and alsoprovides a summative assessment of other peers' feedback according tocertain criteria. These summative assessments can include perceptions,understanding, feedback, and/or other information that occurs betweenthe subjects in the dyad. In one embodiment, the summative assessmentsare collected and analyzed. For example, one or more of the summativeassessments may be converted to scores. In one embodiment, a score maybe calculated according to one or more DLMA algorithms as disclosedherein or according to any other suitable algorithm(s). A pool ofsubjects—such as a class of students—can be divided into one or moregroups having n subjects each. Thus, in this embodiment, each groupconsists of n!/2(n−2)! dyads, where n is the number of subjects. Forexample, a group of six students (i.e. n=6) comprises 15 dyads thatengage in a virtually simultaneous interaction according to anembodiment. Subjects can be assigned to groups randomly, according to amatching algorithm determined by a system coordinator such as aninstructor, or according to an algorithm selected by one or moreapplications being executed on an electronic device that is associatedwith a DLMA system. According to various embodiments, after a task hasbeen completed, a new task may be assigned to the existing groups (i.e.the groups are held static) or to new groups that have been re-matchedfor the pool of subjects. The ensemble of these dyadic interactionswithin a peer group (the DLMA treatment), can then be repeated which mayresult in self-regulating learning and success metrics.

Illustrative DLMA Workflow

FIG. 4 illustrates a method of implementing a DLMA workflow according toan embodiment of the present invention. In embodiments, the method 400shown in FIG. 4 is used to implement the workflow schema of a DLMA PeerAssessment Information System (PAID) as shown in FIG. 5. The method 400shown in FIG. 4 will be described with respect to the electronic device200 shown in FIG. 2. In embodiments, the method 400 may be performed byone or more of the devices shown in system 300 in FIG. 3. For example,one or more electronic devices 320-340 may perform all or a portion ofthe method 400 of FIG. 4 in accordance with embodiments of the presentinvention.

The method 400 beings in block 410 when a pool of subjects are dividedinto groups. For example, referring to FIG. 2, the electronic device 200may receive a name for each of the subjects. In an embodiment, theelectronic device 200 randomly divides the subjects into groups. Inanother embodiment, the electronic device 200 receives inputs thatindicate which group each subject should be in. Thus, in thisembodiment, the subjects are manually placed in groups by a user of theelectronic device 200. In some embodiments, information regarding thesubjects, group sizes, other constraints, group divisions, and/or otherinformation may be received over a network. For example, referring toFIG. 3, tablet computer 320 may receive a list of subjects from webserver 350 through network 310. In this embodiment, the web server mayquery a database to determine the list of subjects. The tablet computer320 may divide the subjects into groups and send information back to theweb server 350 indicating which group each subject should be associatedwith. Numerous other embodiments are disclosed herein and othervariations are within the scope of this disclosure.

The pool of subjects may be divided into groups in any number of ways.In one embodiment, the pool of subjects are manually divided intogroups. For example, an administrator of a task or another personauthorized by the administrator of the task may divide the pool ofsubjects into groups. In another embodiment, the pool of subjects isdivided into groups based on a DLMA algorithm or another algorithm. Oneor more computers can be used to divide the pool of subjects into groupsaccording to embodiments of the present invention. For example, the poolof subjects may be randomly divided into groups.

In one embodiment, the number of subjects that can be assigned to agiven group is determined by an administrator of a task. For example, ateacher may determine that each group should have eight students. Inanother embodiment, the number of subjects assigned to a given group isdynamically determined. For example, referring to FIG. 3, web server 350may determine the number of subjects that can be assigned to a givengroup based on predefined settings, the number of subjects, receivedinput, and/or other factors.

Referring back to method 400, once the subjects are divided into groups410, the method 400 proceeds to block 420. In block 420, the groups aregiven a task. In one embodiment, each group is given the same task. Forexample, each group may be assigned an article to read and an essay towrite about the article. In another embodiment, one or more groups aregiven different tasks. For example, if there are three groups, groups 1and 2 may be given a first assignment and group 3 may be given a secondassignment. As another example, if there are three groups, group 1 maybe given a first assignment, group 2 may be given a second assignment,and group 3 may be given a third assignment. In one embodiment, one ormore assignments may be given manually such as by an administrator ofthe assignment(s). In another embodiment, one or more assignments may beprovided electronically. For example, referring to FIG. 3, web server350 may send an assignment to tablet computer 320. In one embodiment,one or more assignments are selected by an electronic device 200randomly. For example, a database may contain a plurality of availableassignments and the electronic device 200 may query the database todetermine one or more assignments. In another embodiment, an assignmentmay be chosen based at least in part on past performance of one or moresubjects within a given group. Thus, if each subject in a groupperformed well on a previous assignment, then the group may be assigneda more difficult task. Numerous other embodiments are disclosed hereinand variations are within the scope of this disclosure.

Referring back to method 400, once the groups have been given a task420, the method 400 proceeds to block 430. In block 430, the subjects inthe group(s) complete the task and the subjects submit essays regardingthe task. For example, referring to FIG. 3, a subject of a particulargroup may complete the task assigned to that particular group and writean essay using desktop computer 330 regarding the task. In thisembodiment, the subject may submit the essay to the web server 350through network 310. In one embodiment, each subject for each groupsubmits a separate essay. In another embodiment, a subset of thesubjects for each group submits a separate essay. In some embodiments,if a subject does not submit an essay, then a particular value isassigned to that subject for that task. For example, a value of “0” maybe assigned to a subject that does not submit an essay according to oneembodiment.

Referring back to method 400, once the subjects in the group(s) completethe task 430, then the method 400 proceeds to block 440. In block 440,the subjects review and rank the essays submitted by other subjects intheir group and provide textual feedback. For example, referring to FIG.3, a subject in a group may provide rankings for the essays of othermembers of their group and/or textual feedback through an onlinewebsite. In this embodiment, if the subject is using tablet computer320, then the subject may be able to provide the rankings and textualfeedback through the tablet computer 320. The tablet computer 320 maycommunicate with web server 350 through network 310 to send and receiveinformation regarding the task, other subjects in the group, rankings,feedback, and any other necessary or useful information.

In one embodiment, each subject of a group provides rankings and textualfeedback for every other subject in the group. For example, if a groupcomprises eight subjects, then each subject ranks the other sevensubjects from best to worst and provides textual feedback to the sevensubjects. In another embodiment, each subject of a group providesrankings and textual feedback to a subset of the other subjects in thegroup. Thus, in an embodiment, if a group comprises twenty-one subjects,then each subject may provide rankings and textual feedback to ten ofthe twenty other subjects. In one embodiment, the other subjects forwhich a particular subject is to provide rankings and textual feedbackis selected randomly. In other embodiments, the other subjects for whicha particular subject is to provide rankings and textual feedback isselected purposely based at least in part on previously-receivedcriteria, previous results for the group, previous results for one ormore subjects, and/or other information. In one embodiment, a subjectproviding rankings and feedback for another subject in a group may notknow the author of the essay for which rankings and feedback are beingprovided. In another embodiment, a subject providing rankings andfeedback for another subject in a group may know the author of the essayfor which rankings and feedback are being provided.

Referring back to method 400, once the subjects in the group(s) rank theessays and submit textual feedback 440, the method 400 proceeds to block450. In block 450, the subjects submit feedback evaluation for thetextual feedback received. For example, referring to FIG. 3, a subjectin a group may receive the ratings and textual feedback provided byother subjects in the group through a website. In this embodiment, ifthe subject is using tablet computer 320, then the subject may be ableto receive the rankings and textual feedback through the tablet computer320. The tablet computer 320 may receive the rankings and textualfeedback by communicating with the web server 350 through network 310.Similarly, the subject may submit feedback evaluation regarding thetextual feedback received using the tablet computer 320. For example, asubject may be presented with a form to fill out regarding the textualfeedback received from the other subjects which can be completed andsubmitted to web server 350 through network 310 by using the tabletcomputer 320.

Referring back to method 400, once the subjects submit feedbackevaluation for the textual feedback received 450, the method 400proceeds to block 460. In block 460, scores for the subjects arecalculated. For example, referring to FIG. 3, web server 350 maycalculate scores for all or a subset of the subjects. A score for asubject may be calculated in any number of ways. Illustrative models forcalculating various scores are described below in the Illustrative ScoreGeneration Models section.

Referring back to method 400, once the scores for the subjects have beencalculated 460, the method 400 proceeds to block 470. In block 470, allor a portion of the blocks described above with respect to method 400are repeated. For example, if new groups will be formed, then the method400 may be repeated beginning with block 410. As another example, if thesame groups will be maintained, then the method 400 may be repeatedbeginning with block 420.

Preconditions

In one embodiment, a DLMA method complies with the following validitypreconditions if a summative assessment ranking mode is selected.

In an embodiment, the observed within-group distribution of the averagescores based on ranking summative assessment of essays approximates thelatent distribution of the quality of essays within a peer group.

In an embodiment, the observed within-group distribution of the averagescores based on ranking (relative-scale, or forced-distribution)summative assessment of textual feedback approximates the latentdistribution of the quality of verbal feedback within a peer group.

In an embodiment, the observed within-group distribution of the sum ofthe average scores for essay and verbal feedback based on rankingapproximates the latent distribution of the current level of competencywithin a peer group.

In an embodiment, the observed pool-wide distribution of the sum of theaverage scores for essay and verbal feedback based on rankingapproximates the latent distribution of the current level of competencyin the pool of subjects.

In an embodiment, over a series of tasks, the observed pool-widedistribution of the cumulative sum of the average scores for essay andverbal feedback based on ranking approximates the latent distribution ofthe terminal level of competency in the pool of subjects.

In one embodiment, a DLMA method complies with the following validitypreconditions if a summative assessment rating mode is selected.

In an embodiment, the observed within-group distribution of the averagescores based on rating summative assessment of essays approximates thelatent distribution of the quality of essays within a peer group.

In an embodiment, the observed within-group distribution of the averagescores based on rating (absolute-scale, Likert scale, etc.) summativeassessment of textual feedback approximates the latent distribution ofthe quality of verbal feedback within a peer group.

In an embodiment, the observed within-group distribution of the sum ofthe average scores for essay and verbal feedback based on ratingapproximates the latent distribution of the current level of competencywithin a peer group.

In an embodiment, for a given task, the observed pool-wide distributionof the sum of the average scores for essay and verbal feedback based onrating approximates the latent distribution of the current level ofcompetency in the pool of subjects.

In an embodiment, over a series of tasks, the observed pool-widedistribution of the cumulative sum of the average scores for essay andverbal feedback based on rating approximates the latent distribution ofthe terminal level of competency in the pool of subjects.

In other embodiments, one or more of the validity preconditionsdescribed above does not need to be met. In yet another embodiment, noneof the validity preconditions described above are required. In addition,variations of the preconditions described above are within the scope ofthis disclosure.

Illustrative Score Generation Models

The following score generation models described below are illustrativescore generation models and, for simplicity, are described with respectto students in classroom. The models, however, may be used in numerousother contexts. Numerous variations to the models described below aredisclosed herein and variations are within the scope of this disclosure.Those of ordinary skill in the art will realize that the followingdescription is illustrative only and is not intended to be in any waylimiting.

Model 1

In the embodiment of Model 1, a class of N students work independentlyon a single common assignment or project requiring a submission of anessay. In this embodiment, N is generally a relatively small number suchas 6 or below; however, larger numbers are within the scope of thisdisclosure. In the embodiment of Model 1, the rankings of essays areselected from a continuum of most satisfactory to least satisfactory oranother suitable ranking. In this embodiment, each student's essay iscollected, or otherwise submitted, and distributed anonymously among theother students in the class. Thus, in the embodiment of Model 1, eachessay is distributed to (N−1) students for review and every student inthe class has to read, review, and assess everyone else's essay in theclass without knowing the identities of the authors.

After reviewing all of the other students' essays, each student ranks orotherwise orders each essay (other than the student's own essay). In oneembodiment, the student submits a ranking of each student's essay amongthe other students' essays. Thus, the “best” essay (according to thestudent's evaluation) may receive a ranking of “1” and the “worst”ranked essay may receive a ranking of (N−1). In an embodiment, thestudent also submits textual qualitative feedback commenting on theoverall quality of each subject's essay. In this embodiment, theidentify of the author of the feedback is not revealed to the recipientof the feedback. In other embodiments, however, the author of thefeedback is revealed to the recipient of the feedback.

After the feedback from the students have been submitted, each studentreceives back everyone else's feedback to the student's essay. Thus, inan embodiment, a student receives (N−1) pieces of feedback regarding theessay that the student submitted. The student then reviews the feedbackand submits a ranking for each individual feedback. For example, a “1”may be given to the “most helpful and professional” feedback and (N−1)may be given to the “least helpful and professional” feedback.

Suppose, according to an embodiment, that there are N students in aclass indexed i={1, 2, . . . , N}. In this embodiment, a student iranks, or otherwise orders, (N−1) other students' essays so that the“best” gets the rank of 1 and the “worst” gets the rank of (N−1). Inthis embodiment, a student i does not rank-order his/her own essay amongothers. In such an embodiment, a matrix of ranks of essays produced bythe class (scores given are in rows) can be specified as:

$A_{N \times N} = {\left\lbrack a_{ij} \right\rbrack_{N \times N} = \begin{bmatrix}\begin{matrix}N & a_{12} \\a_{21} & N\end{matrix} & \ldots & \begin{matrix}a_{1N} \\a_{2N}\end{matrix} \\\vdots & \ddots & \vdots \\\begin{matrix}a_{N\; 1} & a_{N\; 2}\end{matrix} & \ldots & N\end{bmatrix}}$

where a_(ij) denotes a rank given by a student i to a student j for theessay (or, symmetrically, received by a student j from a student i).

In this embodiment, a_(i)=[a_(i1) a_(i2) . . . a_(ij) . . . a_(iN)] is arow vector of ranks given by student i to all other students such that

$\quad\left\{ \begin{matrix}{a_{ij} = {{N\mspace{14mu} {if}\mspace{14mu} i} = j}} \\{{a_{ij} \in {\left\{ {1,2,\ldots \mspace{14mu},{N - 1}} \right\} \mspace{14mu} {if}\mspace{14mu} i}} = j} \\{a_{i\; 1} \neq a_{i\; 2} \neq \ldots \neq a_{ij} \neq \ldots \neq a_{iN}} \\{a_{ij} = {{N\mspace{14mu} {if}\mspace{14mu} E_{j}} = 0}}\end{matrix} \right.$

where E_(j) is the indicator function such that

$E_{j} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu} {essay}\mspace{14mu} {was}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} {student}\mspace{14mu} j} \\0 & {{if}\mspace{14mu} {essay}\mspace{14mu} {was}\mspace{14mu} {not}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} {student}\mspace{14mu} {j.}}\end{matrix} \right.$

Thus, in an embodiment, a student i does not give a rank to him/her-selfor to a student who did not submit the essay, each of the students needto be ranked (or otherwise ordered) by the student i, and the student icannot give two students the same rank.

Similarly, matrix of ranks of feedbacks produced by the class is (scoresgiven are in rows):

$B_{N \times N} = {\left\lbrack b_{ij} \right\rbrack_{N \times N} = \begin{bmatrix}\begin{matrix}N & b_{i\; 2} \\b_{2i} & N\end{matrix} & \ldots & \begin{matrix}b_{iN} \\b_{2N}\end{matrix} \\\vdots & \ddots & \vdots \\\begin{matrix}b_{N\; 1} & b_{N\; 3}\end{matrix} & \ldots & N\end{bmatrix}}$

subject to the data integrity constraints:

$\quad\left\{ \begin{matrix}{b_{ij} = {{N\mspace{14mu} {if}\mspace{14mu} i} = j}} \\{b_{ij} \in {{\left\{ {1,2,\ldots \mspace{14mu},{N - 1}} \right\} \mspace{14mu} {if}\mspace{14mu} i} \neq j}} \\{b_{i\; 1} \neq b_{i\; 2} \neq \ldots \neq b_{ij} \neq \ldots \neq b_{iN}} \\{b_{ij} = {{N\mspace{14mu} {if}\mspace{14mu} F_{j}} = 0}}\end{matrix} \right.$

where F_(j) is the indicator function such that

$E_{j} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu} {essay}\mspace{14mu} {was}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} {student}\mspace{14mu} j} \\0 & {{if}\mspace{14mu} {essay}\mspace{14mu} {was}\mspace{14mu} {not}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} {student}\mspace{14mu} {j.}}\end{matrix} \right.$

According to an embodiment, if C is the maximum score for the essay(i.e. C is given to an essay that received the rank of 1) and if anessay that received the rank of (N−1) receives the score of 1 and if anyessays that were not submitted receive a score of 0, then the rank ofa_(ij) may be transformed into a score c_(ij):

$c_{ji} = {1 + {\left( {N - a_{ij} - 1} \right)\; \frac{C - 1}{N - 2}}}$

or, equivalently,

$c_{ji} = {{a_{ij}\; \frac{\left( {1 - C} \right)}{N - 2}} + \frac{{C\left( {N - 1} \right)} - 1}{N - 2}}$

For example, if N=6 students and C=5 points, then a transformation ruleaccording to one embodiment may be:

Rank a_(ij) Score c_(ij) 1 5 2 4 3 3 4 2 5 1 Not submitted (6) 0

Similarly, if D is the maximum score for feedback (i.e. D is given to apiece of feedback that received the rank of 1) and a failure to submitfeedback results is given a score of 0, then a transformation rule forrank b_(ij) into the score d_(ij) according to one embodiment is:

$d_{ji} = {1 + {\left( {N - b_{ij} - 1} \right)\frac{D - 1}{N - 2}}}$

or, equivalently,

$d_{ji} = {{b_{ij}\frac{\left( {1 - D} \right)}{N - 2}} + \frac{{D\left( {N - 1} \right)} - 1}{N - 2}}$

Therefore, in some embodiments, C and D reflect relative weights givento the scores for the essay and feedback in the total grade for theassignment.

In one embodiment, the matrix of the individual received essay scoresfor the entire class is (scores received are in rows):

$C_{N \times N} = {{A_{N \times N}^{\prime}\frac{\left( {1 - C} \right)}{N - 2}} + \frac{{C\left( {N - 1} \right)} - 1}{N - 2}}$

In an embodiment, the matrix of the individual given-received feedbackscores for the entire class is (scores received are in rows):

$D_{N \times N} = {{B_{N \times N}^{\prime}\frac{\left( {1 - D} \right)}{N - 2}} + \frac{{D\left( {N - 1} \right)} - 1}{N - 2}}$

According to one embodiment, a student i's grade for the essay is theaverage score received from all his/her peers, who submitted theirfeedback, ideally (NM. Hence, in an embodiment, the column vector ofgrades for essays is:

$\overset{\_}{c} = \frac{C_{N \times N}1^{\prime}}{\sum\limits_{j = 1}^{N - 1}F_{j}}$

where Σ_(j=1) ^(N-1)E_(j)≦N−1, and 1_(1×N)=[1 1 . . . 1] is the rowvector of ones.

In one embodiment, the grade for the essay of a student i is

${\overset{\_}{c}}_{i} = \frac{c_{i_{1 \times \; N}}1^{\prime}}{\sum\limits_{j = 1}^{N - 1}F_{j}}$

where C_(i) _(1×N) is the row vector of essay scores received by thestudent i.

Similarly, the column vector of grades for feedback according to oneembodiment is:

$\overset{\_}{d} = \frac{D_{N \times N}1^{\prime}}{\sum\limits_{j = 1}^{N - 1}G_{j}}$

In an embodiment, the grade for the feedback of a student i is:

${\overset{\_}{d}}_{i} = \frac{d_{i_{1 \times \; N}}1^{\prime}}{\sum\limits_{j = 1}^{N - 1}G_{j}}$

where d_(i) _(1×N) is the row vector of essay scores received by thestudent i,G_(j) is the indicator function such that

$G_{i} = \left\{ \begin{matrix}{1\mspace{14mu} {if}\mspace{14mu} {evaluation}\mspace{14mu} {of}\mspace{14mu} {feedback}\mspace{14mu} {was}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} a\mspace{14mu} {student}\mspace{14mu} j} \\{0\mspace{14mu} {if}\mspace{14mu} {evaluation}\mspace{14mu} {of}\mspace{14mu} {feedback}\mspace{14mu} {was}\mspace{20mu} {not}\mspace{14mu} {submitted}\mspace{14mu} {by}\mspace{14mu} a\mspace{14mu} {student}\mspace{14mu} j} \\{{G_{j} = {{0\mspace{14mu} {if}\mspace{14mu} E_{j}} = {0\mspace{14mu} {\begin{pmatrix}{{{if}\mspace{14mu} a\mspace{14mu} {student}\mspace{14mu} {did}\mspace{14mu} {not}\mspace{14mu} {submit}\mspace{14mu} {essay}},} \\{{she}\mspace{14mu} {should}\mspace{14mu} {not}\mspace{14mu} {evaluate}\mspace{14mu} {feedback}}\end{pmatrix}.}}}}\;}\end{matrix} \right.$

According to one embodiment, the total grade received by a student i is:

p _(i) = c _(i) + d _(i),

In one embodiment the vector of total grades for the assignment of theentire class is:

p= c+ d.

Model 2

The embodiment of Model 2 is an extension of Model 1. In Model 2,instead of a single common assignment as described above with respect toModel 1, the class is given several sequential assignments indexed by k.In this embodiment, the calculations described for Model 1 repeat Ktimes producing matrices A_(k) _(N×N) , B_(k) _(N×N) , C_(k) _(N×N) ,D_(k) _(N×N) , where k={1, 2, . . . , K}.

In the embodiment of Model 2, the vector of total grades for theassignment k of the entire class is:

p _(k) = c _(k) + d _(k),

In the embodiment of Model 2, the vector of total grades for the entirecourse (all assignments) of the entire class is:

p=Σ _(k=1) ^(K) p _(k)=Σ_(k=1) ^(K) c _(k)+Σ_(k=1) ^(K) d _(k).

Model 3

The embodiment of Model 3 is an extension of Model 1. In Model 3, aclass consists of several (L) groups of an approximately equal sizeN_(i); groups are indexed by l. For example, in one embodiment, L isselected such that N_(l) is 6. In another embodiment, L is selected suchthat N_(l) is a number greater than 6. In yet another embodiment, L isselected such that N_(l) is a number less than 6. Therefore, in variousembodiments, L can be selected to be any suitable number.

In the embodiment of Model 3, the calculations discussed above withrespect to Model 1 are performed for each of L groups (replacing N withN_(l)), producing matrices A_(l) _(N×N) , B_(l) _(N×N) , C_(l) _(N×N) ,D_(l) _(N×N) where l={1, 2, . . . , L}.

In the embodiment of Model 3, the vector of total grades for theassignment of the entire class is:

${p = {\begin{bmatrix}{\overset{\_}{c}}_{1} \\{\overset{\_}{c}}_{2} \\\vdots \\{\overset{\_}{c}}_{L}\end{bmatrix} + \begin{bmatrix}{\overset{\_}{d}}_{1} \\{\overset{\_}{d}}_{2} \\\vdots \\{\overset{\_}{d}}_{L}\end{bmatrix}}},$

Therefore, in the embodiment of Model 3, the column vectors of grades ofeach group c _(l) are stacked into a “tall” column vector of grades forthe entire class.

Model 4

The embodiment of Model 4 comprises a hybrid of Model 2 and Model 3. InModel 4, the class is given several sequential assignments indexed byk={1, 2, . . . , K} (with all assumptions of Model 2). In addition, foreach assignment, the class (of size M) is divided into L groups of thesize of N_(l) indexed by l={1, 2, . . . , L} (with all assumptions ofModel 3); M=Σ_(L) ^(l=1)N_(l). Furthermore, in Model 4, for each of theassignments, students are divided into groups randomly, so that for eachassignment a student is given a new random group of peers. Finally,specific projects given to groups may be the same for the entire classor individual for each group; in any case, student within each groupwork on the same group-specific project (independently, i.e. with nocollaboration within the group).

In the embodiment of Model 4, for each assignment k, a student ireceives a grade p_(ki), based on calculations described in Model 2,thus:

p _(ki) = c _(ki) + d _(ki).

In the embodiment of Model 4, the row vector of the student i's gradesfor all assignments is:

p _(i) =[p _(1i) p _(2i) . . . p_(Ki)].

In the embodiment of Model 4, the student i's total grade is:

p _(i)=Σ_(k=1) ^(K) p _(ki) =p _(i)1_(1×K)

Relationship Between Models 1-4

Referring now to FIG. 6, this figure depicts a logical relationship ofalgebraic models of a DLMA score generation process according to anembodiment of the present invention. In the embodiment shown in FIG. 6,Model 1 comprises N subjects, a single group, and a single assignment.Therefore, Model 1 may be appropriate to use in a small class and shortcourses. In FIG. 6, Model 2 comprises N subjects, a single group, and Kassignments. Thus, Model 2 may be appropriate for use in small classesand long courses. In FIG. 6, Model 3 comprises N subjects, L groups, anda single assignment. Therefore, Model 3 may be appropriate for largeclasses and short courses. In the embodiment shown in FIG. 6, Model 4comprises N subjects, L groups, and K assignments. Thus, Model 4 may beappropriate for large classes and long courses.

Model 5

The embodiment of Model 5 is an extension of Model 4. In Model 5, peers'essays are ranked, or otherwise ordered, based on several specifiedcriteria indexed by u={1, 2, . . . , U}. In the embodiment of Model 5,criteria are assumed to be the same for all assignments. However,variations of the present invention in which criteria are different forone or more assignments is within the scope of this disclosure. In theembodiment of Model 5 peers' feedback is ranked based on severalcriteria indexed by v={1, 2, . . . , V}. In various embodiments, Models1, 2 and 3 can be extended in a similar fashion to utilize multiplecriteria for grading.

In Model 5, for an assignment k, for a given group l of the size N_(l),the matrix of ranks of essays based on a criterion u is:

$A_{{ulk}_{N \times N}} = {\left\lbrack a_{ulkij} \right\rbrack_{N \times N} = \begin{bmatrix}0 & a_{{ulki}\; 2} & \ldots & a_{{ulk}\; 1\; N} \\a_{{ulk}\; 2i} & 0 & \ldots & a_{{ulk}\; 2N} \\\vdots & \vdots & \ddots & \vdots \\a_{{ulkN}\; 1} & a_{{ulkN}\; 2} & \ldots & 0\end{bmatrix}}$

where a_(ulkij) is the rank given by a student i to the essay of astudent j in a group l on an assignment k based on an essay criterion u.

In one embodiment, Matrix B_(vlk) _(N×M) is defined similarly, withb_(vlkij) being the rank given by a student i to the feedback of astudent j in a group/on an assignment k bases on a feedback criterion v.In this embodiment, the matrices of scores for each criterion u and v,C_(ulk) _(N×N) and D_(ulk) _(N×N) respectively, can be defined asdescribed in the Model 1, assuming that the maximum possible score isthe same for all criteria. According to one embodiment, the matrices ofscores aggregating all criteria for a group l and assignment k aredefined as weighted averages of the matrices of scores for individualcriteria:

$C_{{lk}_{N \times N}} = \frac{\sum\limits_{u = 1}^{u}{C_{{ulk}_{N \times N}}w_{u}}}{\sum\limits_{u = 1}^{u}w_{u}}$$D_{{lk}_{N \times N}} = \frac{\sum\limits_{v = 1}^{v}{D_{{vlk}_{N \times N}}z_{v}}}{\sum\limits_{v = 1}^{v}w_{u}}$

where w_(u) is the weight of a criterion u in the essay grade and z_(v)is the weight of criterion v in the feedback grade.

In the embodiment of Model 5, the column vector of grades for agroup/for essays in an assignment k is:

${\overset{\_}{c}}_{lk} = \frac{c_{{lk}_{N \times N}}1^{\prime}}{\sum\limits_{i = 1}^{N - 1}E_{i}}$

In the embodiment of Model 5, the column vector of grades for agroup/for feedback in an assignment k is:

${\overset{\_}{d}}_{lk} = \frac{d_{{lk}_{N \times N}}1^{\prime}}{\sum\limits_{i = 1}^{N - 1}F_{i}}$

In the embodiment of Model 5, the vector of total grades for theassignment k of the entire class is:

$p_{k} = {\begin{bmatrix}{\overset{\_}{c}}_{1k} \\{\overset{\_}{c}}_{2k} \\\vdots \\{\overset{\_}{c}}_{Lk}\end{bmatrix} + {\begin{bmatrix}{\overset{\_}{d}}_{1k} \\{\overset{\_}{d}}_{2k} \\\vdots \\{\overset{\_}{d}}_{Lk}\end{bmatrix}.}}$

Thus, in this embodiment, the column vectors of grades of each group c_(l) are stacked into a “tall” column vector of grades for the entireclass.

In the embodiment of Model 5, the total grade received by a student ifor an assignment k is:

p _(ki) = c _(ki) + d _(ki).

In the embodiment of Model 5, the row vector of the student i's gradesfor all assignments is:

p _(i) =[p _(1i) p _(2i) . . . p _(Ki)].

In the embodiment of Model 5, the student i's total grade of for theentire course (that is, for all assignments) is:

p _(i)=Σ_(k=1) ^(K) p _(ki) =p _(i)1_(1×K)

Here, an assumption has been made that all assignments have the sameweight. If all assignments do not have the same weight, then weightingcoefficients can be added to the equation (e.g., by replacing the vectorof 1s, 1_(1×K), with the vector of assignment weights).

Variations

Variations of the score generating processes described above and/or thevarious models described above are within the scope of the presentdisclosure. For example, according to one embodiment, data integrityassumptions a_(i1)≠a_(i2)≠ . . . ≠a_(ij)≠ . . . ≠a_(iN) andb_(i1)≠b_(i2)≠ . . . ≠b_(ij)≠ . . . ≠b_(iN) may be relaxed. Such anembodiment can allow each essay and feedback to be rated rather thanranked. As another example, the random allocation of subjects to groupsmay be replaced with non-random allocation to groups. In such anembodiment, more complex scoring approaches may be used such as higherscoring students being placed in the same group to intensifycompetition.

In one embodiment, the identity of the subject authoring an essay and/orthe identify of the subject providing rankings and/or feedback for anessay is provided. In such an embodiment, one or more DLMA methods maybe used to assess individual contributions to group projects. In anotherembodiment, a dyad of peers may be formed within an open network. Forexample, group randomization may be replaced with network randomization.Thus, in a class of 12 students, dyads may be formed based on the schemashown in FIG. 7 which depicts an example dyad formation in closed groupsand on networks. Numerous other embodiments and variations are disclosedherein and other variations are within the scope of this disclosure.

General

While the methods and systems herein are described in terms of softwareexecuting on various machines, the methods and systems may also beimplemented as specifically-configured hardware, such asfield-programmable gate array (FPGA) specifically to execute the variousmethods. For example, embodiments can be implemented in digitalelectronic circuitry, or in computer hardware, firmware, software, or ina combination thereof. In one embodiment, a device may comprise aprocessor or processors. The processor comprises a computer-readablemedium, such as a random access memory (RAM) coupled to the processor.The processor executes computer-executable program instructions storedin memory, such as executing one or more computer programs for editingan image. Such processors may comprise a microprocessor, a digitalsignal processor (DSP), an application-specific integrated circuit(ASIC), field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The foregoing description of some embodiments of the invention has beenpresented only for the purpose of illustration and description and isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Numerous modifications and adaptations thereof will beapparent to those skilled in the art without departing from the spiritand scope of the invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, operation, or other characteristicdescribed in connection with the embodiment may be included in at leastone implementation of the invention. The invention is not restricted tothe particular embodiments described as such. The appearance of thephrase “in one embodiment” or “in an embodiment” in various places inthe specification does not necessarily refer to the same embodiment. Anyparticular feature, structure, operation, or other characteristicdescribed in this specification in relation to “one embodiment” may becombined with other features, structures, operations, or othercharacteristics described in respect of any other embodiment.

What is claimed is:
 1. A method comprising: receiving, from each of aplurality of subjects, an essay authored by that subject; receiving,from each of the subjects, an essay ranking and a textual feedbackstatement for each of a respective subset of the received essays, eachessay ranking and each textual feedback statement corresponding to oneof the essays authored by one of the subjects; receiving, from each ofthe subjects, a feedback ranking for each of a respective subset of thereceived feedback statements, each feedback statement in that respectivesubset corresponding to the received essay authored by that subject;calculating, for at least one of the subjects, a grade for that subjectbased at least in part on the received essay ratings corresponding tothe received essay authored by that subject and the received feedbackratings for the feedback statements for that subject.
 2. The method ofclaim 1, further comprising: dividing a pool of subjects into at leasttwo groups, one of the groups comprising the plurality of subjects, eachgroup having approximately a same number of subjects; and assigning, foreach of the groups, a respective task requiring that each subject inthat group author a respective essay.
 3. The method of claim 2, whereinthe respective task for each of the groups is a same task.
 4. The methodof claim 2, wherein the respective task for a first group in the atleast two groups is a different task than the respective task for asecond group in the at least two groups.
 5. The method of claim 1,wherein the grade for a particular subject in the at least one subjectis based at least in part on two or more tasks.
 6. The method of claim1, wherein the respective subset of the received essays for a particularsubject comprises each of the received essays except for the receivedessay authored by that particular subject.
 7. The method of claim 1,wherein the respective subset of the received essays for a particularsubject comprises the received essay authored by that particularsubject.
 8. The method of claim 1, wherein an author of an essay in therespective subset of the received essays for a particular subject isunknown to that particular subject.
 9. The method of claim 1, whereinthe respective subset of the received feedback statements for aparticular subject comprises each of the received feedback statementscorresponding to the received essay authored by that subject.
 10. Themethod of claim 1, wherein the received essay rankings for therespective subset of the received essays for a particular subjectrepresent a continuum from a most satisfactory essay to a leastsatisfactory essay.
 11. The method of claim 1, wherein calculating, forat least one of the subjects, a grade for that subject comprisescalculating a vector of total grades for the plurality of subjects. 12.A non-transitory computer-readable medium comprising program code for:receiving, from each of a plurality of subjects, an essay authored bythat subject; receiving, from each of the subjects, an essay ranking anda textual feedback statement for each of a respective subset of thereceived essays, each essay ranking and each textual feedback statementcorresponding to one of the essays authored by one of the subjects;receiving, from each of the subjects, a feedback ranking for each of arespective subset of the received feedback statements, each feedbackstatement in the respective subset corresponding to the received essayauthored by that subject; and calculating, for at least one of thesubjects, a grade for that subject based at least in part on thereceived essay ratings corresponding to the received essay authored bythat subject and the received feedback ratings for the feedbackstatements for that subject.
 13. The non-transitory computer-readablemedium of claim 12, further comprising program code for: dividing a poolof subjects into at least two groups, one of the groups comprising theplurality of subjects; and assigning, for each of the groups, arespective task requiring that each subject in that group author arespective essay.
 14. The non-transitory computer-readable medium ofclaim 13, wherein the respective task for each of the groups is a sametask.
 15. The non-transitory computer-readable medium of claim 13,wherein the grade for a particular subject in the at least one subjectis based at least in part on two or more tasks.
 16. The non-transitorycomputer-readable medium of claim 12, wherein the respective task for afirst group in the at least two groups is a different task than therespective task for second group in the at least two groups.
 17. Thenon-transitory computer-readable medium of claim 12, wherein therespective subset of the received essays for a particular subjectcomprises each of the received essays except for the received essayauthored by that particular subject.
 18. The non-transitorycomputer-readable medium of claim 12, wherein the respective subset ofthe received essays for a particular subject comprises the receivedessay authored by that particular subject.
 19. The non-transitorycomputer-readable medium of claim 12, wherein calculating, for at leastone of the subjects, a grade for that subject comprises calculating avector of total grades for the plurality of subjects.
 20. A systemcomprising: a network; a plurality of electronic devices incommunication with the network; and a server in communication with thenetwork, the server comprising a memory, a network interface, and aprocessor in communication with the memory and the network interface,the processor configured for: receiving, from each of a plurality ofsubjects and from one or more of the electronic devices in communicationwith the network, an essay authored by that subject; receiving, fromeach of the subjects and from one or more of the electronic devices incommunication with the network, an essay ranking and a textual feedbackstatement for each of a respective subset of the received essays, eachessay ranking and each textual feedback statement corresponding to oneof the essays authored by one of the subjects; receiving, from each ofthe subjects and from one or more of the electronic devices incommunication with the network, a feedback ranking for each of arespective subset of the received feedback statements, each feedbackstatement in the respective subset corresponding to the received essayauthored by that subject; and calculating, for at least one of thesubjects, a grade for that subject based at least in part on thereceived essay ratings corresponding to the received essay authored bythat subject and the received feedback ratings for the feedbackstatements for that subject.